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City- wide Systems of Charter Schools: 

Proceed With Caution 

Executive Summary 

The charter school movement dates from the late 1980s and became a popular 
education reform tool in the 1990s. Some 3,000 charter schools exist today. Charter 
advocates made a number of claims regarding how charter schools would improve 
education. The charter, or contract, would free the school of many state and local district 
rules and regulations. Liberated from these onerous bureaucratic rules, charter schools 
would be more efficient and would also offer parents more choices. Most of all, the 
fundamental promise of charter schools was that they would improve achievement. 

The City of Buffalo, New York has proposed to establish a network of charter 
schools under the aegis of the district school board. This appears to be the first time a 
large urban area has made such a proposal. This report addresses what the research 
literature to date has said about the perfonnance of charter schools. It reviews the record 
of charter schools in the context of six questions asked in the planning document, 
Creating a Network of Charter Schools in Buffalo, prepared by the Education Innovation 
Consortium, a Buffalo think-tank, at the direction of the Buffalo School Board. 

What Does Chartering Bring to Buffalo School Reform? 

The planning document contends that charter schools would offer “true school- 
based decision making” and “curricular freedom subject to state perfonnance standards.” 
This report observes that the state performance standards in New York already constrain 
curricular freedom and school-based decision making, and that the No Child Left Behind 
law restricts them further. It also notes that Educational Management Organizations 




(EMOs) could well be the operators of Buffalo charters, and that these organizations, 
having developed their own curricula, offer no school-based freedom at all. 

Can Chartering Bring Higher Levels of Accountability for Results? 

Early charter advocates proposed a strict rule forjudging charter schools: if they 
did not improve achievement, their charters should be revoked. They would be subject to 
performance-based accountability rather than the usual compliance-based accountability. 
In practice, very few charters have been terminated for academic reasons. Most charter 
revocations derive from the mishandling of funds or other financial problems. 

Can Chartering Provide More Quality Choices for Parents? 

This report observes that in evaluations throughout the nation, the evaluators have 
remarked on how similar the charter schools are to the traditional public schools that they 
have replaced. 

Can a Network of Charters Promote the Transformation of the Entire System? 

A hope of charter advocates has been that charter schools would be “laboratories 
of innovation,” and that the innovations begun in charter schools would create change 
throughout the district. The evaluations to date have not documented any such 
transformation. Often there are no mechanisms for dissemination of ideas or best 
practices, and in some cases the teachers and administrators at traditional public schools 
resent the presence of charter schools. In turn, charter school operators have not seen 
dissemination as one of their roles. 

How Do Charters Compare Academically? 

Again, the first and foremost promise of charter schools was to improve 
achievement. To date, no such improvement has been observed. Evaluations from 
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Michigan, California, and Ohio find that charter schools are at best matching the 
performance of demographically similar traditional schools. This brief provides an 
extensive review of state and national evaluations. 

Can Charter Schools Provide Adequate Services to Children With Special Needs? 

While there are some charter schools that serve special needs children, the 
remainder has a smaller number of special education students than the public schools. In 
addition, the special education students that charters do serve generally have mild 
disabilities. Finally, those charters that are market-oriented — those managed by EMOs — 
have fewer special needs students than charters that are not market-oriented. 

Recommendations 

Based on the research data available to date on charter schools, it is recommended 
that the Buffalo School District, local and state legislators, and other education policy 
makers undertake the following tasks: 

1 . Critically evaluate the research and conclusions suggested in Creating a 
Network of Charter Schools in Buffalo. That document omitted many 
evaluation studies and uncritically — and sometimes erroneously — accepted the 
conclusions of studies it did include. 

2. Ensure that agencies proposing charter schools to the school board specify in 
clear and measurable tenns what they expect to accomplish. Many evaluations 
to date report only vaguely stated goals. 



Page 3 of 4 




3. Ensure that the procedures the charter school operators will use to attain the 
goals stated in the charter are defined in measurable terms that will provide 
clarity as to whether or not the goals have been met. 

4. Ensure that all parties are clear about the meaning of statements of 
accountability, and the consequences of not meeting the goals as stated in the 
charter. 

5. Present charter schools as an asset to existing schools rather than a threat. 

To date, charter schools have had little influence on the operations of 
traditional public schools, and some charters have been resented. If the district 
goal is transformation of the district schools, charter schools must be seen as a 
welcome addition to public education, and there must be clearly defined 
mechanisms by which charter schools contribute to the improvement of non- 
charter schools in the district. 

6. Ensure that proposals offer specific and adequate statements about the 
administrative, personnel, and fiscal skills of those who will be in charge of 
managing the school. Many charter schools, and not only those whose charters 
have been revoked, have had personnel and financial struggles. 

7. Ensure that proposals have specific statements about their approach to special 
education, and that this approach is consonant with district goals. 

8. Obtain the services of an external, disinterested party to track the progress of 
charter schools and to conduct an evaluation prior to the opening of any charter 
schools under the proposed Network. 
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City- wide Systems of Charter Schools: 

Proceed With Caution 

Gerald Bracey 
George Mason University 

Introduction 

Like most education reforms since Sputnik was launched in 1957, charter schools 
sprang from perceived problems in the public schools. The idea of a charter was that the 
school would enter into an agreement with some agency that would free the school from 
many district and state regulations. In return for this increased autonomy, charter schools 
promised to increase achievement. Charter advocates anticipated some ancillary benefits 
as well: innovations, improved efficiency, and increased program choices for parents. In 
addition, some proponents predicted the transformation of the public schools, since these 
would be forced to compete for students with the charter schools. 

First proposed by a teacher in Vermont in the 1970’s, charter schools became 
popular after receiving attention in the late 1980’s from Albert Shanker, then president of 
the American Federation of Teachers. 1 Not long after Shanker touted the charter concept 
in a speech in Minneapolis, the Minnesota legislature passed the first charter law in 1991 
and the first charter school, City Academy Charter School, opened in St. Paul in 1992. 
Charter schools have been characterized as “everyone’s reform.” Left-leaning organizers 
of free schools and the alternative school movements of the 1970s saw them as a natural 
extension of that effort; right-leaning advocates of vouchers and a privatized system of 
education saw charter schools as a first step toward their ultimate goal. 




The charter movement grew rapidly. In the 1993-94 school year, there were 34 
charter schools; by 1998, 1,000; and, currently, about 3,000. Yet the promotion of 
charter schools as an instrument for reforming school districts has declined in the decade 
since 1994. Even during the period of most rapid growth, charter schools were solely 
dependent on individual effort; no authorizing agency initiated any systematic effort to 
install charter schools in a given state or district or series of districts. In fact, Shanker, 
somewhat disillusioned with large-scale reform efforts, believed that working to refonn a 

•5 

single school would be the strength of the movement. Even where there were many 
charter schools managed by the same Educational Management Organization (EMO), 
such as in Michigan, those organizations took the initiative and approached the 
authorizers, not the other way around. 

The Buffalo Proposal 

Now, however, a new element has been added: a larger, urban school district, 
Buffalo, New York, is planning to develop a "network" of charter schools, the number as 
yet unspecified, with the city public school system as the authorizer. This would mark 
the first time a district has attempted to use charter schools in such a systemic way: not 
only to provide an alternative educational setting for the students directly enrolled in that 
setting, but also to incite change throughout the entire district. Its equivalent can be seen 
only in those districts that established groups of magnet schools. The Buffalo initiative is 
the most ambitious and therefore the most momentous of any charter school effort to 
date, and has the potential to serve as a model for other large, urban districts. Charter 
schools have been under the research microscope almost since their inception, and the 
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amount of evidence compiled has also grown exponentially, particularly in the period 
since 1998. The likelihood that Buffalo’s effort could set a tone for further school refonn 
nationwide warrants evaluating the district’s plan against research evidence garnered 
from existing studies of charter schools. 

The Buffalo plan grew out of conversations between the Education Innovation 
Consortium (EIC) and members of the Buffalo Board of Education. The Board invited 
the EIC to submit a “concept paper,” which the EIC did in March 2003 in a document 
called The Renaissance School Project. On March 26, 2003, the Board authorized the 
EIC to develop a feasibility study for the ideas described in the February paper. This 
study was delivered to the Board as Creating a Network of Charter Schools in Buffalo in 
October, 2003, and unanimously adopted by the Board on December 3. The Board has 
issued a Request for Concept Proposals, due by March 15, 2004; as of February 26, 

15 inquiries had been received, and proposals are expected to follow most inquires. The 
Board will then ask some or all organizations that submitted concept proposals to deliver 
full proposals. 



Questions and Answers 

EIC’s report Creating a Network of Charter Schools in Buffalo asked a series of 
eight questions and sought to answer them. 4 The plan of this brief is to examine the 
research and evaluation evidence as it pertains to six of those eight questions, as well as 
the answers EIC provided. Two of the questions are not addressed in this report: 
Questions 6 and 8, for which little evidence exists, and for which answers can only be 
highly speculative. 
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1. What does Chartering Bring to Buffalo Public Schools Reform? 



The EIC report answers this question with the traditional promise of autonomy for 
performance, a tradeoff that has been the hallmark of the charter school movement. In 
addition, the EIC report claims that charters offer “true school-based decision-making” 
and “curricular freedom subject to state perfonnance standards” (p. 9-11). 

As long as the provisions of the No Child Left Behind legislation are in force, 
freedom is greatly restricted at the elementary and middle school level. No Child Left 
Behind requires the establishment of state standards and tests aligned with those 
standards. All children in grades 3-8 must be tested every year in reading and math and, 
soon, in science. Schools that fail to make a pre-determined amount of progress on the 
tests are subject to sanctions, with the sanctions becoming more severe for each 
consecutive year of failure. New York, in particular, has a long history of strong 
statewide Regents examinations. 

Recent research by the Council of the Great City Schools suggests that curricular 
freedom and school-based decision-making may be harmful to urban districts. The 
Council commissioned MDRC (a non-partisan public policy research group in New York 
City, formerly called The Manpower Demonstration Research Corp.) to study urban 
districts that were making relatively rapid progress in improving achievement and those 
that were not. The rapidly improving districts did not all look alike, but they shared a 
focus on instructional coherence. 5 
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Urban districts employ less qualified and often less experienced teachers than 
suburban districts. Building-level curriculum choices might not be appropriate; indeed, 
building-level choices might be the more difficult for charter schools. Without the 
support normally provided by various specialists in the central office, all aspects of 
administration, personnel, and finance must be managed by the school. In effect, each 
school becomes a separate district and in some states is treated as such. The difficulties 
inherent in running a school have created significant opportunities for the EMOs. 
Schnaiberg reported instances where operators of “mom and pop” charter schools 
turned the keys over to an EMO, saying, “It was a burnout situation.” 6 As a 
consequence of this lack of experience, in a period of five years, the percentage of 
charters operated by EMOs rose from 17 percent to 74 percent. 7 

A similar phenomenon has been observed in Ohio. A report from the Legislative 
Office of Education Oversight (LOEO), which monitors Ohio charter schools, found that 
Ohio “community schools” (Ohio’s name for charter schools) benefited greatly from the 
assistance provided by management companies. Community schools not operated by a 
management company were at a disadvantage, because they were responsible for all 
aspects of running the schools, “...ranging from curriculum design to staff hiring and 

o 

evaluations to planning budgets.” 

The irony of this situation is that schools run by EMOs offer no curricular 
freedom or building-level decision-making whatsoever. Indeed, evaluators of charter 
schools in Michigan referred to such charters as “cookie-cutter schools.” 9 Depending on 
the location of an EMO’s headquarters, fees paid to the EMO might even leave the state. 
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2. Can Chartering Bring Higher Levels of Accountability for Results? 



Again, this is the fundamental promise of charter schools: we will abandon 
accountability defined by compliance in favor of accountability defined as performance. 
Charter school champion Joe Nathan of the University of Minnesota put it this way: 

Hundreds of charter schools have been created around this nation by educators 
who are willing to put their jobs on the line to say, “If we can’t improve student 
achievement, close down our school.” This is accountability — clear, specific and 
real. 10 

A similar statement comparing charter school accountability to the more 
traditional kind of accountability in public schools was put forth by Paul Hill and 
colleagues at the University of Washington: 

Conventional public schools are considered accountable because they 
must follow all rules set by local and state school boards, and abide by all 
the provisions of contracts that these boards enter with unions and other 
organizations. Charter schools are exempted from many of these rules and 
instead are required to demonstrate student learning. 11 

Although the improve-or-close philosophy is discussed often, it does not appear to 
play much of a role in the closing of charter schools. Of the 154 charter schools reported 
closed in an October, 2002, report from the Center for Education Refonn (CER), only 13 
were closed for academic reasons (the report had not yet documented the reasons for an 

additional 40 closures not included in the 154 figure). Most charter revocations derive 

12 

from the mishandling of funds or other financial problems. 
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Based on the data from CER (which in fact promotes charter schools as an 
educational reform tool), charter schools have earned a closure rate of 6.7 percent. There 
is no consensus as to how to interpret this low closure rate; it is sometimes presented as 
evidence of the movement’s success and sometimes as evidence of the failure of 
accountability in charter schools. The reported closure rate may understate matters, 
however. The CER report lists another 77 charter schools that were re-absorbed into the 
district and 84 schools that received a charter “but never opened and thus are not counted 
as closed” (p. 2). If these last two categories are placed with the closures in a category 
that might be called “unsustained,” then the failure rate of charters is 12.4 percent, not 
6.7 percent. 

3. Can Chartering Provide More Quality Choices for Parents? 

According to charter proponents, charter schools would serve as 
laboratories of innovation. As Nathan put it, “The charter school movement 
attempts to promote widespread improvement in public education both by 
allowing people to create new kinds of schools and by encouraging existing 
school systems to improve in order to compete effectively with these new 
schools.” ~ Ted Kolderie, another Minnesotan and long-time charter advocate, 
emphasized the “second-order effect” of instigating change that he expected 
charter schools to have on the educational system. 14 

Thus, the quality choices question is twofold: will charter schools develop 
innovations? And, will these innovations create change in the districts? The answer to 
these questions thus bears on question 7 in the Buffalo plan as well as question 3. 
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7. Can a Network of Charters Promote Transformation of the Entire 



System? 

In evaluations of charter schools reported to date, neither aspect of innovation 
appears as a dominant theme. In a survey of numerous states, Eric Rofes, then of Policy 
Analysis for California Education, expressed surprise at the lack of innovation: 

Certain innovations and changes in school districts and traditional public schools 
[that I hypothesized would take place] had rarely occurred: Few superintendents, 
principals and teachers in district schools were thinking of charter schools as 
educational laboratories or attempting to transfer pedagogical innovations from 
charters to the district schools; districts were still building large school facilities 
and rarely creating smaller schools; the large urban districts studied rarely had 
responded in meaningful ways to charter laws and charter schools . 15 
An analysis of charter schools in California by Amy Stuart Wells, then at UCLA, 
offered an explanation of Rofes’ observations: “There are no mechanisms in place for 
charter schools and regular schools to leam from each other.” As a consequence “all but 
two of the public school educators, and nearly all of the educators we interviewed, 
reported that they had very little infonnation about what was going on in the charter 
schools, and nearly all of the educators we interviewed said they saw little if any direct 
impact of charter schools on their schools .” 16 

The recent RAND evaluation of charter schools in California examined whether 
or not public schools, in reaction to charter schools, had restructured teacher 
compensation, restructured hiring or firing practices, changed their curricula, changed 
instructional practices, or changed professional development. The RAND team reached a 
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conclusion similar to that of Wells and Rofes: “[Our analyses] suggest that conventional 
public schools have not felt much of a competitive effect from charter schools and have 
not changed their operational practices significantly .” 17 

Other reports suggest that even had there been a mechanism for dissemination, 
there was little to disseminate. When Michael Mintrom of Michigan State University 
examined Michigan charters, he concluded: “More striking is the degree of similarity that 
we find across all schools, be they charter or traditional.” Mintrom solicited statements 
about innovations that had been made in public and charter schools. He then prepared a 
list of such innovations and asked principals to rank the degree of innovation — without 
telling the principals what kind of school had adopted the innovation. He concluded that, 
“while some charter schools are definitely doing some innovative things, overall 
Michigan’s charter schools are no more remarkable than many traditional public schools 
in their practices .” 19 

Mintrom found that charter schools exceed public schools in the adoption of 

innovations in only three areas: non-traditional scheduling, mandatory foreign language 

20 

study, and uniforms." 

Horn and Miron reached a similar conclusion in their study of Michigan “public 
school academies” (PSAs), another name for charter schools in that state: “Based upon 
school visits and documentation provided by the PSAs, we conclude that there are limited 
innovations being developed and applied in the PSAs. In fact, the charter schools were 
remarkably similar to the regular public schools, with the notable exception of generally 
smaller student enrollments, the presence of additional adults in the classroom, 
governance, and the span of contracted services.” Within the schools operated by 
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EMOs, Horn and Miron found great similarity; as noted earlier, they referred to such 

22 

schools as “cookie cutter” schools. " EMOs operate more than 75% of all charter schools 



in Michigan. 23 

When asked to list their innovative practices, teachers and administrators in 
Michigan charter schools brought forth the following: 

• Community activity experiences for students 

• Time set aside for reading 

• Multiage classrooms 

• Before- and after-school programs 

• Individual Education Plans for all students 

• Small class size 

• Greater Individualization 

• Teaching assistants and volunteers in classrooms 

• Cooperative learning 

• Chicago Math and Saxon Math 

• Learning Labs 

• Foreign-language instruction 

• Outcomes-based education 

• Direct Instruction 

• Montessori methods 

• Curriculum with specific focus — for example Native American and African 

24 

American culture, arts, ecology, etc. 
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All of these practices are well known to public school educators and do not 
qualify as innovations. These practices might very well increase the range of choices 
available to parents in a given locale, however. Horn and Miron did not investigate this 
possibility. Some would challenge, however, whether or not all of the choices listed 
above represent “quality” programs or instruction. 

The similarity between charter schools and traditional public schools takes on 
increased importance when one considers that charter schools are often presented as near- 
revolutionary tools. For instance, the Buffalo charter plan document quotes Tony 
Wagner of Harvard as saying: “Reforming our present system isn’t the solution. We 
need to reinvent it. This study [of charters in Buffalo] explores an approach that is 
significantly closer to reinvention.” Or again: “Despite its leadership’s laudable and 
persistent efforts at refonn during the last several years, BPS carries all the baggage 
common to Industrial Era school systems across the country. ” The report does not spell 
out the nature of the “baggage” these schools carry, but the implication is that charters are 
schools for the information age, and that there is an enormous difference between charter 
schools and traditional public schools. The descriptions above from Michigan and 
California are not consonant with that conception. 

In fact, another evaluation of Michigan charter schools, by investigators at 
Michigan State University, concluded that charter school directors who planned 
significant innovations risked eliciting reactions from parents ranging from skeptical to 
hostile (“not-with-my-kid-you-don’t”). Parents seemed to feel that they were satisfied 
with the schools they themselves had attended and they wanted the same for their 
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children. “Insofar as this is what parents want, charters have little to gain and much to 

97 

lose from experimentation with innovative practices.” 

4. How Do Charters Compare Academically? 

Charter school advocates have described a number of advantages that charter 
schools would have over traditional public schools, including more innovation and 
greater efficiency. The most often cited advantage, in some cases the raison d’etre of 
charter schools, is that they would increase achievement. 

The remarks quoted in the response to question 2 illustrate what charter 
advocates have promised for their institutions. Two years after the first charter 
school opened, Jeffrey Henig, observed that little was being done in the way of 

systematically looking at charter schools. He noted that charter schools and 

28 

authorizers had no inclination towards such an examination." 

In 1996, former assistant U.S. secretary of education, Chester Finn, and 
two colleagues, Louann Bierlein and Bruno Manno, examined accountability 
plans in charter states and reported that no state had developed a strong evaluation 

9Q 

system for evaluating its charter schools. 

Three years later, Manno pointed to three reasons that accountability 
information has been lacking for charter schools. One is lack of data; two is 
underdeveloped accountability systems; and three - perhaps the cause of the first 
two - is that charter authorizers and operators are not interested in developing 

30 

rigorous accountability agreements. 
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The most recent comment on the situation, from 2003, indicates by its title that 
information on achievement is still lacking: Student Achievement in Charter Schools: 
What We Know and Why We Know So Little , 31 

Still, there have been some systematic evaluations, some of which appeared too 
late to be included in the Miron and Nelson summary. The most systematic studies have 
occurred in Arizona, California, Michigan, and Ohio. There have been, as well, a few 
attempts to examine achievement in charter schools in state and national studies. 

Arizona’s charter law was enacted in 1994. Initially, the Arizona Department of 
Education had an office of charter school evaluation. When it began uncovering 
problems in charter schools, however, the state superintendent, who, as a state legislator, 
originally wrote Arizona’s charter law, shut the evaluation office down and withheld its 
report for a year. This produced a series of semi-scandalous articles in the local and 
national media. " The state superintendent, Lisa Keegan, declared, “In the main, I’m 
pleased, far and away with the quality of the public charter schools”; but Marilynn 
Henley, who had headed the evaluation team and then entered the private sector, 
contradicted her: “I resent her [Keegan] and others who say charter schools are just like 

33 

or no worse than district schools. Charter schools are way worse.” 

The following year, 1999, the Morrison Institute for Public Policy at 
Arizona State University observed the lack of strong analyses of student 
achievement in charter schools, despite the fact that improving achievement is a 
central goal of Arizona’s charter school law. 34 

The Institute’s own general analysis concluded that although rates of 
improvement were similar for charter and public school students in elementary 
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school, charter school students began to lag behind public school students in 
middle school. The disparity increased in high school, with a difference of 10-15 

35 

points between charter and public school students. 

The analysis could not determine if charter schools were selecting lower 
achieving students or if the students were not learning as much. As will be seen later, 
similar lags appeared in California middle and high schools. 

Taking advantage of Arizona’s student data base, which permits the 
tracking of students from year to year as long as they are in Arizona public 
schools, the Center for Market-Based Education at the Goldwater Institute 
purported to find a significant advantage in reading, though not in mathematics, 
for students who attended charter schools for two or three years as compared to 
students who attended traditional public schools (TPSs) for the same period of 

■ • 36 

tune. 

The study is an econometric analysis, and it is difficult to know precisely what 
kind of analysis the Goldwater researchers performed. After reviewing the study, 
Douglas Harris, an economist at the Economic Policy Institute, said: “It’s hard to tell 
what they actually did. Their descriptions of the methods are really poor.” Christopher 
Nelson of Western Michigan University and Kevin Hollenbeck of the W. E. Upjohn 
Institute for Employment Research critiqued the study on a number of methodological 
grounds. Their report also contains a number of conditional clauses, such as “As we 
understand it” and “it appears as if,” suggesting that they, too, are uncertain about the 

38 

methodology. 
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It’s important to note, however, that even if the study is taken at face value, 
although the researchers claim effect sizes are large, the advantage for charter schools is 
small: the “points” mentioned in this statement are percentile ranks. The gains cited 
would translate into only about one-half test item per year. 

Early researchers on charter schools in California were hampered in their 
evaluations of achievement by the fact that the California assessment program was in a 
transition period and different tests were used in different years in different districts. 
Nonetheless, two reports found that evaluating outcomes was difficult because of how the 
goals were specified. According to SRI International, “many of these charters were 
written in the early days of the law, and few had experience writing or evaluating them. 
Many charters included goals in their charters that are difficult to measure. For example, 
many charter schools included goals such as the appreciation of cultural differences, the 

39 

empowerment of staff and parents, stronger community ties, or improved social skills.” 
Similarly, Wells’ evaluation of California charters recognized that the “goals and 
outcomes are often vaguely written and ill-defined; they frequently cover a wide range of 
desired outcomes, such as the goal of ‘enabling pupils to become self-motivated, 
competent, and life-long learners.’” 40 

The state’s current STAR assessment program has been in place for over four 
years. Several evaluations of California charters using STAR data appeared in 2003. 
Investigators from the RAND Corporation compared traditional schools, traditional 
schools that had converted to charter status, and charter schools that had started from 
scratch. 41 After examining test score results from 1997-98 through 2001-02, the 
researchers concluded: 
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Our analysis suggests that charter schools generally have comparable or slightly 
lower test scores than do conventional public schools. Achievement, however, 
varies by type of charter school. Conversion schools that deliver their instruction 
in classrooms had mixed results, with some scoring the same, higher, or lower 
than conventional public schools. Start-up schools using classroom instruction 
had slightly higher test scores in everything but elementary math, where the 
scores are slightly lower. Conversion or start-up schools that deliver at least a 
portion of their instruction outside the classroom had lower test scores across the 
board. 42 

The RAND group cautioned, however, that the students in schools with instruction 
delivered outside of classrooms might differ in important ways from those in 
conventional classrooms, be they traditional or charter schools. RAND had no means of 
identifying any such differences. The RAND researchers also found some school districts 
that had the capability of tracking individual students over time, offering the possibility 
of longitudinal rather than cross-sectional analyses. In these data, “the analysis shows 
that charter school students are keeping pace with comparable students in conventional 
public schools.” 43 

Charter advocates, though, promised to exceed the performance of public schools, 
not merely to keep pace with it. 

A statewide analysis of California’s test-based Academic Progress Index (API) 
found results similar to those in the RAND study. David Rogosa conducts an annual 
analysis of API changes for the California Department of Education. In 2003 he 
examined such changes for 1998-99 through 2001-2002. 44 Rogosa’s results follow: 
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Table 1: Changes in the Academic Progress Index 1998-99 THROUGH 2001-02 





1999 


2002 


Change 


Grades 2-6 


Charter 


588 


664 


76 


Regular 


620 


694 


74 


Grades 2-8 


Charter 


598 


657 


59 


Regular 


622 


685 


63 


Grades 9-11 


Charter 


597 


611 


15 


Regular 


617 


637 


20 



In more targeted analyses, Rogosa finds advantages for regular public schools in 
grades 7 and 8. The advantage is magnified when the data are analyzed according to 
socioeconomic status. While elementary charter schools with high percentages of 
economically disadvantaged kids enjoyed a slight edge over traditional public schools, 
the public schools gained dramatically more than charter schools at the middle grades. 
Indeed, the “gain” for disadvantaged 8 th graders in charter schools was -12.1 points. 

Margaret Raymond of the Hoover Institution found results similar to those that 
Rogosa observed, breaking out the data in different categories. Raymond compared 
traditional public schools, charter schools, and what she termed “local competitor” 
schools, these last being traditional public schools in a district that also contained at least 
one charter school. This category would seem to be of limited utility or validity given the 
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prior comments on the charters’ lack of impact on regular schools, and indeed Raymond 
does not appear to have made use of the distinction in reporting her findings. 

Elementary schools in all three groups were comparable. Charter schools gained 
half as much as other public schools at the middle grades, a finding that accords with 
Rogosa’s, but Raymond contended that charter high schools gained twice as much as 
other public high schools. Her analysis is statistically flawed, however. Raymond chose 
the school as the unit of analysis rather than the pupil; this gives small schools more 
weight than they deserve. Raymond’s procedure is comparable to finding an average 
SAT score by using the state-level results for, say, New York and North Dakota. In 
2003, 145,562 seniors in New York took the SAT, while 394 in North Dakota did so. 

But an analysis that starts with the state average gives both states equal weight. To get 
the true average, the average in New York should be multiplied by the number of 
students in New York taking the SAT. A similar calculation needs to be made for North 
Dakota. The two resultant figures then need to be divided by the total number of students 
taking the test in both states combined. When this is done, Raymond’s findings at the 
high school level are in line with those of Rogosa. 

In Michigan, as we have seen, evaluators have found charters to closely resemble 
regular public schools. Michigan has had a long-standing statewide testing program, the 
Michigan Educational Assessment Program (MEAP), which provides much of the 
achievement data for evaluations. Three of the evaluations have examined the 
achievement of charter schools relative to public schools. Jerry Horn and Gary Miron 
observed that over a four-year period, the passing rates for charter schools declined while 
those for public schools in the host districts of the charter schools rose. The variability of 
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pass rates was considerable, but typically favored the host district. The rates and ranges 
of the rates are in Table 2. 



Table 2 45 


Assessment 


4-year Average 


Range of Pass Rates 




Charter 


Host 

District 


Charter 


Host 

District 


4 th -grade math 


45.2% 


63.9% 


33-35% 


49-68% 


4 th -grade reading 


35.8% 


51.7% 


33-38% 


38-55% 


5 th -grade science 


20% 


30% 


14-22% 


19-37% 


5 th -grade writing 


43% 


56% 


36-57% 


49-69% 


7 th -grade math 


33.8% 


47.3% 


25-36% 


32-51% 


7 th -grade reading 


30.5% 


37.9% 


19-34% 


33-44% 


8 th -grade science 


9% 


18% 


8. 6-9. 4% 


15-20% 


8 th -grade writing 


46% 


59% 


37-62% 


50-72% 



Eric Bettinger of Columbia University concluded: “When charter schools are 
compared to public schools with similar characteristics, pupils in charter schools score no 
higher, on average, and may even be doing worse.” 46 Randall Eberts and Kevin 
Hollenbeck of the Upjohn Institute for Employment Research found charter students 2-4 
percentage points lower on the MEAP 4 th -grade reading, 4 percentage points lower on the 
MEAP 5 th -grade science and 6 percentage points lower on the MEAP 5 th -grade writing 
test. 47 

It is worth noting that these three evaluations approached the data differently and 
used different analytical methods, and still concurred in their conclusions. 

One additional piece of research comes from Gary Miron and Christopher Nelson. 
They developed a technique for looking at the relative gains or losses of charter schools 
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compared to other public schools. They subtracted the gains of the charter schools from 
the gains of the host districts and concluded: “[W]ith the exception of 4 th -grade math, all 
differences favor regular public schools.” 48 

As noted earlier, some have viewed with alann the dominance of EMOs in 
Michigan charter schools. When Miron and Nelson calculated the relative gains of EMO 
schools for non-EMO schools, they found the non-EMOs to produce larger gains. 49 

More skepticism attended the arrival of charters in Ohio than in most other 
states. Journalists early on questioned the motivation of those who advocated 
them 

To determine if the charter schools, permitted only in the seven largest 
metropolitan areas of Ohio, would improve over time, the Ohio legislature established the 
Legislative Office of Education Oversight (LOEO), which began issuing reports in 1999 
on various aspects of the charter experience. The LOEO’s final report summarizes the 
achievement of charter schools, known in Ohio as “community schools,” relative to other 
public schools. Generally, when the LOEO compared traditional schools with the 
community schools, 14 of 20 comparisons were significant, and 13 of 14 favored 
traditional public schools. 

When the LOEO compared the passing rates on the Ohio Proficiency Tests of 
each community school to its demographically matched traditional school, it found mixed 
results. Of the 415 comparisons, 270 were not statistically significant. Of the 145 that 
were significant, 103 favored traditional schools and 42 favored community schools. 
Comparisons using scaled scores from the Ohio Proficiency Tests yielded small effect 
sizes favoring traditional schools. When the comparisons used passing rates rather than 



Page 20 of 29 




scores, the effect sizes were larger and favored traditional schools. In sum, the data 
generally indicated that the charter schools were not performing as well as 
demographically matched public schools. 

These results are somewhat tentative, however, because over the five years 
of LOEO’s reports, many of the charter schools have failed to deliver required 
information, information routinely supplied by public schools. The LOEO noted 
that many community schools failed to report data that allowed them to be 
evaluated according to their charters, although such data are required by law. 

Those that did not report data often did not meet the academic perfonnance goals 
set out in their charters. 50 

Thus, compared to either traditional public schools or to the requirements 
specified in their charters, the community schools come up short. The states reviewed are 
the most active charter states in the nation. These four states contain 1,343, or 44 
percent, of the 2,996 charter schools operating at the start of the 2003-2004 school year. 51 
The sample of charter schools they provide, then, is unlikely to be a non-representative 
sample of all charter schools. The possibility must be considered that charter schools are 
not succeeding as hoped. 

A recent report by researchers from The Manhattan Institute concluded that 
charter schools have a positive effect. The study involved 1 1 states. Perhaps more 
importantly, it examined only schools that served “general” populations rather than 
“targeted” populations. Schools serving advantaged or disadvantaged populations were 
not included unless they received those populations solely as a function of their locations. 
The study excluded “cyber” schools because there was no means of comparing such a 
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school with a nearby traditional public school. Schools that had been ordinary public 
schools and converted to charter status were also excluded because they were not genuine 
schools of choice. 

The results indicated that in math, charter schools out-perfonned nearby 
traditional public schools by an effect size of 0.08, equivalent, according to the 
researchers, of 3 percentile ranks. In reading, the effect size was 0.04, equivalent to 2 
percentile ranks (in both cases, the effect size is that as measured from the 50 th 
percentile). 

Among the 1 1 states, only five had sufficient data to yield state estimates of 
effects. Of these, only Texas showed a consistently significant effect in both reading and 
math. Results were insignificant in Arizona, North Carolina, and California. Results 
were inconsistent in Florida. In Florida, the effect size was significant in math on the 
Stanford Achievement Test, but not on the Florida Comprehensive Assessment Test 
(FCAT). In reading the effect was significant on the FCAT but not the Stanford. In both 
instances of significance, significance was attained at the .05 level. 

5. Can Charter Schools Adequately Provide Services to Children With 
Special Needs? 

The ability of charter schools to provide services to children with special needs is 
a difficult one to analyze. A summary of 12 states finds the percentage of students in 
special education in charter schools consistently smaller, sometimes dramatically smaller, 
than the percentage in regular schools. Connecticut, for instance, has more than 14 
percent of its students in regular schools receiving special education services, but only 2 

53 

percent of the students in charter schools receiving such services. 
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On the other hand, the District of Columbia schools have slightly more students in 
special education than do charter schools within the district. 54 Like all averages for 
special education, however, this average could be misleading. Some schools target 
special education students, skewing the average. For instance, while Miron and Nelson 
found that charter schools had, on average, a smaller percentage of special education 
students than their counterpart public schools, some of the charters did have more than 40 
percent. 55 

Moreover, differently managed charter schools might react to special education 
students differently. Researchers from The George Washington University set out to test 
the hypothesis that charter schools were “creaming” students from public schools — 
taking the academically superior students. They found little evidence of this. They did 
find, however, what they called “cropping.” While market-oriented charter schools 
didn’t include a disproportionate number of superior students, they did tend to exclude 
students at the other end of the academic spectrum, “cropping” high-cost students, 
including special education students, out of the picture. Such charter schools had a much 
smaller percentage of special education students than non-market oriented charter 
schools. 56 

Because it is not clear at this time what kind of entity or entities will be managing 
Buffalo’s charter schools, the market orientation (or absence thereof) of applicants to 
operate schools could be important. Charter schools nationally spend more money on 
administration and less on instruction than regular schools. 57 This result might be 
exaggerated in for-profit EMOs. 
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A national case study of 32 charter schools conducted jointly by Westat and SRI 
International found that most students with disabilities in charter schools had only mild 
disabilities. Parents and students both reported more individualized attention in the 

58 

charter schools, but did not think that the actual services provided there were better. 
Another national study found that charter schools often have difficulty finding 
appropriate special education staff. 59 

Recommendations 

Based on the research data available to date on charter schools, the Buffalo 
School District, local and state legislators, and other education policy makers should 
undertake the following tasks: 

1 . Critically evaluate the research and conclusions suggested in Creating a 
Network of Charter Schools in Buffalo. That document omitted many 
evaluation studies and uncritically — and sometimes erroneously — accepted the 
conclusions of studies it did include. 

2. Ensure that agencies proposing charter schools to the school board specify in 
clear and measurable tenns what they expect to accomplish. Many evaluations 
to date report only vaguely stated goals. 

3. Ensure that the procedures the charter school operators will use to attain the 
goals stated in the charter are defined in measurable terms that will provide 
clarity as to whether or not the goals have been met. 
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4. Ensure that all parties are clear about the meaning of statements of 
accountability, and the consequences of not meeting the goals as stated in the 
charter. 

5. Present charter schools as an asset to existing schools rather than a threat. 

To date, charter schools have had little influence on the operations of 
traditional public schools, and some charter schools have been resented. If the 
district goal is transfonnation of the district schools, charter schools must be 
seen as a welcome addition to public education, and there must be clearly 
defined mechanisms by which charter schools contribute to the improvement 
of non-charter schools in the district. 

6. Ensure that proposals offer specific and adequate statements about the 
administrative, personnel, and fiscal skills of those who will be in charge of 
managing the school. Many charter schools, and not only those whose charters 
have been revoked, have had personnel and financial struggles. 

7. Ensure that proposals have specific statements about their approach to special 
education, and that this approach is consonant with district goals. 

8. Obtain the services of an external, disinterested party to track the progress of 
charter schools and to conduct an evaluation prior to the opening of any charter 
schools under the proposed Network. 
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